Traditionally, relational databases offer a reliable way of storing and retrieving large amounts of data, and are a common type of database. Previously, companies used to install database software on physical computers with large attached storage in the form of hard disks, and other permanent storage media, in conjunction with faster “Random Access Memory” (RAM). In recent years, the volume of data has grown exponentially and companies have realized that this approach is incapable of handing these volumes of data. Now more than ever data is extremely important for businesse sand individuals alike. Accurate and reliable data is crucial to support business decisions so it is imperative to store and query it reliably and quickly. Currently, the manner in which data is stored on server is somewhat inefficient due to speed and access requirements. To this end, to minimize or overcome such challenges, one solution is to scale horizontally using many smaller servers (e.g., nodes) that still simulates a single “centralised” storage service that is familiar to user and developers alike. When data is highly distributed, however, it is difficult to handle system-wide (i.e. global) atomic update to data, especially because nodes may fail randomly (e.g., temporarily or permanently). The network may also fail to deliver key edit instructions to all nodes as required so naive systems that assume reliable messaging will fail in random and untraceable ways. By comparison, single machine databases (i.e., non-distributed) offer features (reliability and consistency) that are difficult to achieve in a large and distributed system of computing “nodes”. The properties of ACID (atomicity, consistency, independence and durability) have become the cornerstone of most enterprise applications and are core features of most traditional relational database engines (e.g., MySQL, Oracle).
To address the above noted drawbacks of databases and storage, the concept of NoSQL (or non-SQL) databases has emerged to describe distributed databases that offer scalability as the expense of some other benefits of traditional relational database management systems (RDBMS). Often these NoSQL systems did not provide“atomic” updates, or supported only a limited data model compared to a traditional relational database. The simplest type is a distributed Key→Value store, where a fixed data “key” could be used to store and then retrieve a value (often just an arbitrary array of bytes).
Some highly distributed systems have since managed to provide ACID properties on top of simple key-value data models. Despite these advancements, achieving a fully ACID level of guarantee over a widely distributed database system is challenging and often the speed of the system is fundamentally limited by the need to maintain the illusion of a globally consistent data model.
In parallel to these above-mentioned developments business systems have, for a variety of regulatory, security and audit reasons increasingly had to support time-stamping of updates. Such systems maintain snapshots of states in a way that the full state of the database at points in the past can be recovered even if some data is later edited and/or deleted. The benefits of this for audit purposes is clear, given that regulations create an increasingly important need for a strongly auditable system.
The time-granularity of systems varies from periodic full dumps (such as a traditional backup) to fully time-transactional systems that record every separate edit and can therefore view a database to any time in the past. Within the stable of conventional RDBMS systems, SQL server introduced the idea of temporal tables, and Oracle provides some support with their “flashback” technology. An optimal time-transactional system is one which records every edit, has unlimited storage, scale and high-performance data querying abilities on a relational data model. In view of all the above, there is a need for efficient methods and systems to overcome the above-mentioned challenges, by providing a system and methods that enable the recording of edits, provide unlimited (or substantially unlimited) storage, is scalable and delivers high performance.